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ABSTRACT 

New modalities for assessing student learning exist as a result of advances in computer 
technology. Conventional measurement practices have been transformed into computer based 
testing. Although current testing replicates assessment processes used in college classrooms, a 
greater opportunity exists to use computer technology to create authentic assessments. 
Assessments that mirror real world scenarios and often teach and enhance skill building while 
measuring student achievement. This case study examines the evaluation of business simulations 
as authentic assessment instruments using the five dimensions of authenticity developed by 
Gulikers, Bastiaens and Kirschner (2004). The paper will conclude with a review of business 
simulations available in the marketplace. 
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INTRODUCTION 



alls for greater accountability in higher education continue to be featured in the news as the cost of a 
college education rises faster than inflation. Amidst the calls for greater accountability, criticisms of 
poorly prepared business graduates resonates throughout business meetings and in business journals. 
The need for students to go beyond recalling facts and to demonstrate reasoning, problem solving ability, and 
creative and innovative approaches to business problems is clear (Shavelson, Baxxter, & Pine, 1991). In the United 
States, several federal laws and government regulations require that colleges and universities demonstrate that they 
are preparing students for employment. Title IV of the Higher Education Act of 1965, along with the Higher 
Education Opportunity Act passed in 2008, address the need for colleges and universities to prepare students for 
gainful employment in a recognized occupation (Epstein, 2010). Higher education professionals are anticipating 
major changes to the law in 2013 with the renewal of the Higher Education Act 


Since the 1990s, there have been numerous advocates of authentic assessment as one of the critical pieces 
necessary to help students learn (Wiggins & McTighe, 1998; Leahy, et al., 2005, Johnson, 2007). New calls for 
accountability in higher education include developing methods and practices for assessing outcomes at the 
institutional, program, and course levels. The focus on skills indicate there is a gap between the skills employers 
identify as being important for first time job seekers and the skills that colleges focus on in the classroom (Gabric & 
McFadden, 2001). In order to address the gap, curriculum should be redesigned to include applied skills, such as 
problem-solving, communication, and analytical skills. Owyang (2010) reports that a survey conducted by the 
Altimeter group found that the number one skill employers wanted for new business graduates was real world 
business experience. Calls for authentic assessment methods have increased in response to the perceived need for 
teaching applied skills to business students. 


Student outcomes assessment continues to receive attention from external constituencies regarding the level 
of skills students can demonstrate in the workplace and across varying type of institutions. Higher education 
researchers and curriculum designers are focusing on authentic assessment techniques and backward design whereby 
curriculum is designed and assessed based on the skills a graduate is expected to demonstrate on their first job after 
earning their degree (Palm, 2008). Authentic assessment requires students to use prior learning, recent studies, and 
relevant skills to solve realistic and often complex problems (DiMartino & Castaneda, 2007). Passing a multiple - 


© 2012 The Clute Institute http://www.cluteinstitute.com/ 


449 




American Journal Of Business Education - July/August 2012 


Volume 5 , Number 4 


choice test or writing an essay demonstrates an understanding of the acquisition of knowledge. These assessment 
techniques fall short of having students demonstrate what they can do with the knowledge learned in a course or 
program. Employers of business graduates are looking for graduates with business experience and proven 
leadership and communication skills (Germain, 2009). 

The traditional static educational model of instructors delivering lectures and assessing student learning 
using multiple-choice and essay exams is designed to measure knowledge, not to have students actually solve 
authentic tasks that solve real-life problems (Gulikers, Bastiaens, and Kirschner, 2004). This paper explores the 
question becomes how to adequately assess whether college graduates have attained the skills and competencies 
they need to become employed in their field of study by examining the historical use of simulations and whether 
simulations can be used for authentic assessment in addition to their use as learning tools. 

COMPUTER BASED SIMULATIONS 

Real world scenarios designed to imitate activities that professionals perform in their everyday jobs are the 
basis for simulations. When designed properly, the simulations require the same type of activities performed in the 
work place. Medical and aviation schools were early adopters of simulation type training to prepare practitioners in 
those fields (Jeffries, 2005). Medical students use simulations to hone their surgical skills on computerized 
mannequins before working with live patients. Spinello and Fischbach (2004) used problem based learning to 
develop an interactive online public health simulation to provide opportunities for public health students to practice 
decision making in a safe environment. Studies have shown that simulations not only improve the efficiency of 
healthcare workers but they are also effective recruiting tools (Heather, 2010; Rieber, Betz, Muth, Nikendei, 
Schrauth, & et ah, 2009). Aviation students receive training in flight simulators to perfect their flight skills prior to 
manning an aircraft. Bell and Waag (1998) point out that simulations to train fighter pilots improved pilot readiness 
for in flight training. 

Simulations and gaming applications, as well as social networking, are gaining acceptance in the higher 
education arena. Increased use of simulations in the non-health and aviation classrooms is evident by the 
proliferation of simulations and educational games available on the internet. Early data from several studies 
supports the value of scenario based learning. Students in a sports management class were asked to compare a 
traditional lecture-based classroom to a classroom utilizing simulations in place of lectures. The results indicated 
that not only did students prefer the learning simulation environment but they also spent more time outside of the 
classroom preparing for the simulation classes than the lecture classes (Newman, Irwin, Klenosky, & Gillentine, 
2003). The addition of the integration of multimedia, including simulations, into the online and on ground college 
classroom can be used to attract and retain the attention of students. Research has shown that when students are 
engaged in what they are studying, they comprehend the material at a deeper, more cognizant level (Tan, Tse, & 
Chung, 2010). Klassen and Willoughby (2003) evaluated learning by simulations by issuing a questionnaire to 
students prior to beginning a simulation on inventory systems and again after the student had played the simulation. 
Most of the results indicated that students did in fact learn by participating in the simulation. In addition, many 
students reported that they not only enjoyed the learning experience but found it a worthwhile endeavor (2003). 

The role of instructors in facilitating simulations effectively in the classroom is to provide feedback to the 
learner and increase repetitive practice. Adapting to various learning strategies is an important factor to the success 
of simulations in the classroom (Spillane, 2006). While previous research supports the idea of using simulations to 
facilitate learning, the authors were interested in whether simulations can be used to actually assess student learning. 

BUSINESS PROGRAMS AND SIMULATIONS 

Business program development occurs during phases with ongoing program maintenance and major 
redevelopment efforts occuring approximately every four years. During 2009, administrators in the college of 
business administration began the program review process and determined that rather than continue with making 
small changes to the current business curriculum, it was time for a major redesign of the business curriculum. One 
problem that had been identified with the existing program was an over-reliance on traditional assessment methods. 
Written projects and multiple-choice assessments were the sole methods of evaluating student learning. In terms of 
Bloom’s taxonomy, the focus was on declarative learning at the lower end of the scale (Nilson, 2003). The 
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assessments used in the program were also outdated. Many of the summative assessments had been developed when 
the program was created in 2006. University Administrators, Advisory Board Members, and the Business Faculty 
wanted to integrate authentic tasks in the program. A focus group of business leaders also emphasized the need for 
business graduates to have real-life problems solving skills. 

Authentic assessment is seen by some researchers as being performance assessments, while others see 
authentic assessment as going beyond just performance tasks and asking students to perform realistic tasks in a 
realistic context (Gulikers, Bastiaens & Kirschner, 2004). In order to determine the best methods for assessing 
student learning, the committee reviewed the competencies and objectives and evaluated various assessment 
methods currently used for their validity as authentic assessment instruments. Multiple choice exams, essays, 
papers, projects, case studies and presentations were currently used to assess student achievement. 

Creating more authentic, real-world tasks as assessments in the business program required looking at 
computer-based testing and business simulations currently available in the marketplace. After the business 
competencies were identified, a faculty committee began the process of identifying and evaluating simulations for 
use as assessments in the program. Textbook companies were contacted along with other suppliers of education 
content. Reviews of simulations used in business training in the corporate world were reviewed. 

THE STUDY 

Before the study could begin, the faculty committee and university administrators finalized the program 
level competencies and objectives that would be demonstrated by business graduates. The faculty committee 
including subject matter experts in accounting and finance, quantitative analysis, marketing, management, business 
law, economics, and information technology, examined the learning outcomes to identify areas where simulations 
may be available to measure student mastery of the competencies. An environmental scan was completed where the 
committee searched for business simulations that were currently available in the marketplace. This qualitative study 
examines the simulations identified using the five dimensions of authentic assessment as identified by Gulikers, 
Bastiaens and Kirschner (2006). 

Guiding the researchers’ study of simulations were four questions: 

1. What types of business simulations are available in the marketplace? 

2. How do the available simulations align with program competencies and objectives? 

3. Can the simulations be used to assess business competence? 

4. How can the evaluation process be improved when evaluating future assessment options? 

Examinations of the faculty committee’s work began with the gathering of evidence from committee 
meetings and correspondence. The examination was twofold: First, each item was analyzed to determine how it 
demonstrated group process. Group interactions were evaluated using the effectiveness criteria established by 
Robbins and Judge (2007) for interacting groups. The level of task orientation, amount of interpersonal conflict, 
commitment to developing a solution, group cohesiveness, and speed of decision-making were examined. Second, 
each document was scored using the five dimensions of authentic assessment checklist provided in Table 1. 

FINDINGS 

An analysis of 192 e-mails regarding the business program development (102 of which pertained directly to 
the simulations) were examined and revealed that the review of simulations began in 2009 and continued into early 
2010. Faculty members working in the College of Business and assessment department staff were asked to identify 
simulations that might be appropriate for use in the revise program. In June 2009, a curriculum committee of seven 
individuals was formed and tasked with summarizing the information gathered and developing a list of possible 
simulations to be used as summative assessments in the business program. 

Over the course of eight months, seventeen off-the-shelf business simulations were examined. The 
simulations examined included a variety of products from several diverse providers. Simulations from publishers. 
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like McGraw-Hill and Pearson, were evaluated as were simulations from technology companies such as Go Ventures 
and CapSim. Individuals on the committee were assigned simulations and asked to perform a cursory review and 
provide feedback to the committee. 

The project manager developed an analysis checklist for examining the selected simulations. Committee 
members were asked to evaluate a series of questions on a scale of 1-5 with 1 meaning the simulation does not 
satisfy the requirement and 5 meaning it completely satisfies the requirement. The committee completed the 
evaluation of simulations using the simulation checklist that was developed. Only one simulation included a final 
comprehensive exam. The items the committee members were to evaluate as they reviewed the simulations for use 
in the business program are presented in Table 2. 

Table 1: Authentic Assessment Frequency Table 

Research Item: _ 

Date:_ 


Dimensions 

Frequency 

Task Criterion 


Meaningfulness and relevance in the students’ eyes 


Degree of ownership of problem and solution 


Criterion at the students’ educational level 


Structure of task is well-defined 


Monodisciplinary/Multidisciplinary 


Physical Context 


Similar to professional work space 


Availability of professional resources 


Similarity to professional time frame 


Social Context 


Individual work/decision-making 


Groups or collaborative work/decision-making 


Result/Form 


Demonstration of competence 


Presentation to others 


Multiple indicators of learning 


Criteria 


Based on criteria in professional practice 


Related to realistic products/processes 


Transparent and explicit 


Criterion-referenced leading to profile score 



Table 2: Ratings Checklist 


Item 

Scale 1-5 

Ease of use 


Ease of administration 


Learning materials 


Alignment of learning resources and tasks in simulation 


Record of outputs 


Detailed feedback on student performance 


Flexible pacing 


Compartmentalized content 


Activities align with program and course objectives 


Reasonable per student costs 



After several general discussions, as evidenced by the e-mail correspondence and the review of the minutes 
of the meetings, six simulations out of the 17 originally reviewed were selected for an in-depth review. Based on 
feedback from committee members, many of the simulations were determined to be inappropriate for a graduate- 
level program, were too complicated to operate, did not produce the warranted outcomes, or were too costly. Those 
simulations were not selected for an in-depth review. The six simulations chosen for further review and the 
committee members’ cumulative scores are presented in Table 3. 
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Table 3: Simulations and Scores for Further Review 


Simulation Provider 

Capsim 

McGraw-Hill 

IntelliSIM 

SimuLearn 

Media Spark 

Espire Learning 

Simulation Name 

Capsim 

Business 
Strategy Game 

Global 

Tycoon 

Virtual 

Leader 

Go Ventures 

Executive 

Challenge 

Average Score: 

3.73 

3.8 

2.43 

3.31 

3.86 

2.9 

Total Score out of 50 

37.3 

38.0 

24.3 

33.1 

38.6 

29.0 


The simulations review process got off to a slow start as the team tried to identify simulations available in 
the marketplace. Web searches revealed few usable simulations. Conversations with colleagues in the College of 
Business and inquiries with assessment specialists, advisory board members, and textbook publisher representatives 
yielded the first list of potential simulations. After a series of presentations by Harvard Business School and other 
presenters, the team refocused their review of simulations with the development of a set of criteria that would be 
used to assess the viability of simulations. 

Each committee member was assigned several simulations to review and report back to the group. This 
process was implemented to speed up the simulations review process. Dividing up the simulations among group 
members was more efficient, but not highly effective in terms of thoroughness of review. It takes several hours and 
often training by the simulation provider in order to ascertain the effectiveness of using a particular simulation. 

One weakness of assigning the simulations to one or two individuals for review was that the team often had 
questions about the simulation that the reviewers had not asked. Also, at one point the vendor had sold other 
members of the team on the usability of their simulation. When the appointed reviewers reported back favorably, 
other members of the team were not convinced of the findings. The simulation had to be reviewed a second time by 
other members of the team. 

The original simulations review process and criteria were adjusted as the review process continued. 
Review of the first few simulations revealed additional considerations that were important in selecting the 
appropriate simulations. Ninety-five pieces of evidence were examined based on the five criteria for authentic 
assessment checklist. The results are shown in Table 4. 


Table 4: Results Based on the Authentic Assessment Frequency Table 

Research Item: _ 


Dimensions 

Frequency 

Task Criterion 

32 

Meaningfulness and relevance in the students’ eyes 

25 

Degree of ownership of problem and solution 

3 

Criterion at the students’ educational level 

13 

Structure of task is well-defined 

11 

Monodisciplinary/Multidisciplinary 

1 

Physical Context 

9 

Similar to professional work space 

1 

Availability of professional resources 

1 

Similarity to professional time frame 

6 

Social Context 

10 

Individual work/decision-making 

11 

Groups or collaborative work/decision-making 

10 

Result/Form 

9 

Demonstration of competence 

4 

Presentation to others 

1 

Multiple indicators of learning 

4 

Criteria 

6 

Based on criteria in professional practice 

8 

Related to realistic products/processes 

11 

Transparent and explicit 

N/A 

Criterion-referenced leading to profile score 

N/A 
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The major focus of the committee was on the tasks that students completed as part of the simulation. 
Educational level was established as a criterion for review early in the process eliminating the need to discuss it 
frequently. Physical aspects of the simulation and experience were the least discussed element of authentic 
assessment. The setting for tasks was defined by the university’s delivery method. There was little discussion of 
physical setting because it was assumed that all students would complete tasks in their homes. Although professional 
time frame was discussed briefly, the committee was focused on providing a learning experience for students rather 
than matching the professional time frame for tasks. The definition of professional space is evolving. Many 
professionals work from home with a desk and computer. E-mail and documents revealed a concern on the part of 
the committee around how to facilitate group work in an online setting. The committee acknowledged that group 
work was an important component of success in business, but communications indicate that the logistics of 
measuring student achievement in group decision-making was difficult in an asynchronous, online classroom. 

Analysis indicates results were included in the simulation evaluation checklist, but e-mail was infrequently 
used to discuss the analysis. This was surprising since results would be used as the summative assessment. 
Researchers believe that because results were included in the evaluation checklist, there was little need to discuss 
results outside of the checklist. The frequency of discussion of criteria elements was low. The researchers believe 
that this element was not discussed because of early assumptions made by the group that all simulations needed to 
produce an experience that mimicked professional practice. 

Simulations are great tools for helping student master knowledge and skills and they work well as 
formative assessments. Completing simulations helps students to check their progress toward mastering workplace 
competencies. There is little evidence that business simulations currently available in the marketplace are structured 
such that they can be used as summative assessments in college courses. Simulations provide computer based 
training and practice opportunities for students; however, the structure of the simulations reviewed did not include 
summative assessment measures for evaluating student learning. 

IMPLICATIONS FOR FUTURE STUDY 

For non-traditional college students with below average computer skills, completing simulations may be 
formidable. Mastery of the computer and the logistics of using simulation software may force students to focus 
more on simulation navigation than on learning and demonstrating competency. Providing technical support for 
students will be critical to student success with simulations. In addition to technology skills, students will also need 
problem sovling skills and critical thinking skills prior to attempting simulation type learning (Petrakou, 2010). 
Simulations require that students demonstrate practical skills in business and although learning takes place during 
the simulation process, most students will need significant background in the course content as well as how to 
perform higher level Bloom’s level skills to successfully pass the simulations. 

Although the five elements of authentic assessment created a framework for evaluating the viability of 
simulations as authentic assessment techniques, there were several additional criteria that the committee judged as 
being important to the selection and use of a simulation. To be a viable assessment tool, simulations would need to 
have a revision schedule and the data elements would need to be changed on a regular basis to deter academic 
dishonesty. Too often, answer keys and copies of tests are circulated on the internet. Simulation costs are another 
concern expressed by the faculty committee. For the university to assess student learning using simulations, the 
costs of using simulations would have to be reasonable. 

Ease of access was an important consideration when evaluating the simulations. The university enrolls 
students from across the United States and it was important that students were able to easily access the simulations. 
Technology support was an important consideration for the business curriculum committee as these simulations 
would be used asynchronously with online students who need accessible technical support. The ability to integrate 
simulations into the university’s learning management system was important. Output results and the timing of output 
was a consideration for students and faculty. The committee considered timely feedback as important to the student 
evaluation and learning processes. 
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University administrators were concerned that simulations costs might increase assessment costs. The 
benefits of using simulations needed to exceed added costs for delivering assessments. Authentic assessments are 
only viable if they can be developed and delivered in a cost effective manner. Estimating costs to develop authentic 
assessments is difficult but research suggests that the direct and indirect costs of developing authentic assessments is 
twice as much as traditional assessment development (Rothman, 1995; Hardy, 1995; Wiggins, 1990), Simulation 
costs were a consideration for the committee. The university enrolls several hundred students in the business 
program and the per-student cost of accessing the simulation and the costs for tailoring the simulation to the 
business learning outcomes were considered by the curriculum committee. 

When evaluating the simulations, the committee found that the simulations were written at many different 
levels. From rudimentary business skills to sophisticated business decisions requiring significant analysis and 
evaluation of information, simulations cover a broad range of academic levels. For committee members assessing 
the level of simulation could be time consuming due to a lack of supplementary materials. Higher level simulations 
provided robust learning materials including readings, supplemental resources, and associated websites. 

A conceptual framework and supporting learning materials are necessary for simulations to support student 
learning at the college level. Simulations allow students to interact with complex systems and ideas but assessing 
the actual learning that takes place can be challenging (Frezzo, Behrens, & Mislevy, 2010). Finding an effective 
instructional model which supports both hard (technical or procedural) and soft (people, communication) skills can 
be challenging as the soft skills are more difficult to quantify. 

CONCLUSION 

Simulations are computer-based tools that provide opportunities for students to practice real-life skills that 
they need to be successful in the workplace (Wiggins, 1990). No longer limited to primarily aviation and medical 
education, the number of simulations and computer based training tools for business practitioners is increasing. 
Using simulations in the online classroom works pedagogically for skill building or formative assessment where 
students can use results of simulations to self assess their progress in mastering real world skills. Authentic 
assessment advocates who argue that conventional measurement practices emphasize passing a test rather than 
demonstration of real world skills would find simulations to be more authentic measurements of student learning 
(Tanner, 2001). The findings in this study do not contradict indications that simulations are more authentic 
assessments of real world skills; however, the findings indicate that simulations are not designed for the sole 
purpose of assessing the competencies of business students. At the present time, off-the-shelf business simulations 
fall short in the measurement of student learning at the summative assessment level. As the number and types of 
simulations and gaming applications for business professionals continue to increase, added features that allow for 
the measurement of student competencies will create more accurate assessment tools and more authentic assessment 
strategies. 
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